Skip to content

feat(memory-pool): cross-machine zero-copy tensor transport for end-edge-cloud collaboration - #3079

Open
tang-canran wants to merge 102 commits into
dora-rs:mainfrom
tang-canran:memory-pool
Open

feat(memory-pool): cross-machine zero-copy tensor transport for end-edge-cloud collaboration#3079
tang-canran wants to merge 102 commits into
dora-rs:mainfrom
tang-canran:memory-pool

Conversation

@tang-canran

@tang-canran tang-canran commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Memory Pool: Zero-Copy Tensor Transport across Daemons and Machines for End-Edge-Cloud Robotics

(1) Design — new architecture for multi-daemon and cross-machine deployment

This PR extends the memory-pool transport to the deployment topologies real edge-cloud robotics actually run: multiple daemons on one host (sensor, perception, and inference pipelines as separate daemons on a robot or edge box) and daemon clusters across machines (edge ↔ cloud). The additions:

Same-host multi-daemon — direct read across daemons

  • The receiver locates the sender's pool segment directly by machine-qualified name (pool_{node_id}_{machine}_{counter}) and reads it in place, bypassing the daemon relay and even the mirror push entirely (same-host direct detection). Native dora has no such path: outputs to a consumer on another daemon are pinned to the daemon relay (Readiness barrier (#2666) counts unreachable remote subscribers: fixed 5 s per-node startup stall + lost direct-zenoh fast path for local subscribers in multi-machine dataflows #2738), so the same topology costs 89 MB/s instead of ~5.8 GB/s (65×).
  • For GPU pools, the sender exports the CUDA IPC handle into the pool header at registration; a same-host cross-daemon receiver imports it once and reads device memory zero-copy (32 GB/s measured).
  • Control-plane notifications (RegisterPool/RegisterPoolAck/FreePool) between daemons ride a zenoh SHM payload for same-host delivery.

Cross-machine — reliable large-frame data plane with GPU staging

  • The data plane uses the daemon's zenoh channel with blocking congestion control: large frames queue and drain over slow WAN links instead of being dropped. Native dora's relay uses Drop + express: fragments are silently dropped when the TX queue (16-batch cap) backs up on a slow link, and the daemon↔daemon data plane hangs at any frame size once the RTT crosses ~200 ms — reproducible on loopback with tc qdisc add dev lo root netem delay 100ms (a 0.5 MiB frame hangs identically to 40 MiB). There is no working native path for WAN large frames.
  • GPU endpoints get CPU staging pools: a pageable transit pool on the send side (GPU_A → DtoH → CPU_A) and a pinned pool on the receive side (CPU_B → HtoD → GPU_B), both registered and reused per frame; the full path GPU→CPU→zenoh TCP→CPU→GPU runs at link bandwidth (~38 MB/s on the 1-Gbps LAN link; GPU staging adds no per-frame allocation).
  • Machine-qualified pool names and the daemon-side memory manager (touched_by / targeted cleanup) prevent cross-machine ID aliasing and clean up pools on every daemon involved — orphan sweeps are scoped to the daemon's own machine so a sibling daemon's live segments are never touched.

Existing single-daemon behavior is unchanged; the event stream, zenoh, and daemon relay code are untouched.

(2) Performance: memory pool vs native dora vs ROS 2 (3 scenarios × 4 device pairs + cross-machine links)

Benchmark: 100 frames of 10000×512 int64 (40.96 MB), turn-based per-frame handshake, Average transfer throughput (data_bytes / (t_received − t_send)), same payload and cadence on both sides. Native dora runs with the 256 MiB zenoh SHM pool configured (its best zero-copy configuration). GPU columns (cpu2cuda/cuda2cpu/cuda2cuda) are measured with the same benchmark; the WAN row is the cross-machine example's 61.44 MB frames (15000×512) on the same data path.

Memory pool (MB/s, this PR):

Scenario cpu2cpu cpu2cuda cuda2cpu cuda2cuda
Single daemon 6157.9 3197.1 6792.7 44145.8 (same-GPU IPC)
Same-host cross-daemon 5787.5 2616.6 6977.6 32691.6 (IPC direct read)
Cross-machine 1-Gbps LAN (5090↔A100, RTT ~0.2 ms) 38.0 37.5 38.3 37.5 (GPU_A→CPU→TCP→CPU→GPU_B)
Cross-machine true WAN (workstation↔A100, RTT 4.7 ms, link ~7 MB/s) ~4

Native dora (MB/s, same benchmark):

Scenario cpu2cpu note
Single daemon 2409.6 zenoh SHM zero-copy active
Same-host cross-daemon 89.3 pinned to the daemon relay (#2738); node-to-node zenoh mesh is same-machine-only
Cross-machine 1-Gbps LAN ~100 relay-bound, ≈82–85% of line rate (independently reproduced: 819 Mbps on the same link)
Cross-machine true WAN no working path Drop + express silently drops fragments on the slow link; the producer blocks on its ack forever (observed with 20–61 MB frames); loopback tc netem delay 100ms reproduces the data-plane hang at any size

Speedup: 2.6× single-daemon, 65× same-host cross-daemon. On a 1-Gbps LAN the native relay already runs at ≈85% of line rate, so that link leaves little headroom (the pool is at 32% of line rate there — docker bridge/NAT and the per-frame handshake are the current limiters, not the link; we report it as-is rather than label it WAN). The claims that matter are the same-host 65× (reproducible on one machine without a network) and the WAN rows below.

Cross-machine landscape — ROS 2 network DDS on a real WAN (RTT 38.3 ms, n=12/size, byte-contract all green):

Payload Raw TCP per-frame (control arm) ROS 2 network DDS ROS 2 vs raw TCP
1 MiB 21.80 MB/s 1.29 MB/s 17.0×
4 MiB 43.74 MB/s 1.19 MB/s 36.9×
16 MiB 68.08 MB/s 1.07 MB/s 63.4×
  • ROS 2 is not bandwidth-limited on the WAN — it is locked to a fixed advance per RTT. Raw TCP scales with frame size (21.8 → 68.1 MB/s, the fixed 38.3 ms RTT amortizing over larger frames), while ROS 2 stays flat at 1.07–1.29 MB/s. 1.1 MB/s × 38.3 ms ≈ 42 KB/RTT: the reliable-transport pacing advances only a fixed chunk per round trip.
  • The memory pool has no such signature. Across the two links measured, pool throughput tracks link bandwidth (38–44.6 MB/s on the 1-Gbps LAN, ~4 MB/s on the ~7 MB/s WAN with 61.44 MB frames at 13–16 s/frame, ≈60% of the link), and blocking congestion control queues and drains instead of dropping fragments.
  • Same-host control (16 MiB, loopback): ROS 2 network DDS 801 MB/s vs dora's cross-daemon relay 107 MB/s — ROS 2's two-hop publisher→subscriber path is 7.5× faster than dora's four-hop node→daemon→daemon→node relay. The relay path is replaceable; the pool removes it for same-host and replaces it with a reliable queueing data plane cross-machine.

(3) Purpose and significance — enabling end-edge-cloud tensor pipelines

VLA models and world models are driving explosive demand for on-device inference [1–4]; end-edge co-computing is becoming the mainstream deployment paradigm, keeping data transport inside the LAN at 1–6 ms round trips versus tens-to-hundreds of milliseconds over the WAN to the cloud [10,11]. A robot pipeline in this paradigm moves tens-of-MB GPU tensors (camera frames, point clouds, VLA features) through several processes — sensor → preprocessing → inference — on the same host, and then to the edge cluster or the cloud.

This PR makes that pipeline zero-copy at every hop:

  • On the end device: tensors stay on the GPU across processes (same-GPU CUDA IPC at 42–44 GB/s; CPU↔GPU at 3–7 GB/s), and the zero-copy read frees the device's limited CPU for inference instead of serialization and relay copies.
  • At the edge (multi-daemon on one host): separate daemons per pipeline stage no longer pay the daemon-relay tax — direct cross-daemon reads deliver 65× over native dora (89 MB/s → 5.8 GB/s), so multi-process edge pipelines scale without the message-passing overhead. (On a 1-Gbps LAN the relay is already ≈85% of line rate, so the cross-machine LAN gain is small by physics; the same-host win is where the edge multi-process tax is.)
  • Edge ↔ cloud (WAN): native dora cannot carry large frames across machines at all (Drop + express drops fragments; the producer blocks forever), and ROS 2's network DDS collapses to ~1.1 MB/s on a 38 ms-RTT WAN — RTT-paced, not bandwidth-bound. The pool's blocking-control relay delivers large frames at link bandwidth (~60% of a ~7 MB/s WAN link, scaling with frame size) — closing the gap where distributed deployments currently have no viable data path for training-data uploads, fleet telemetry, or cloud-assisted inference.

One API (write_memory_pool / read_memory_pool) covers end, edge, and cloud with automatic device- and topology-aware path selection, turning message-passing deployments into shared-state pipelines without changing the dataflow description.

tang-canran and others added 30 commits July 31, 2026 15:40
…ensor transfer

Extend classify_transport with a fifth parameter is_cross_machine.
When true, the function returns NetworkZenohTransport regardless of
GPU topology — data serialises and routes through the daemon's
Zenoh channel for cross-host delivery.

Add 4 cross-machine test YAMLs: cpu2cpu, cpu2cuda, cuda2cpu,
cuda2cuda — each deploys sender on machine A and receiver on
machine B via _unstable_deploy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- DaemonNodeEvent::WriteMemoryPool: node→daemon, carries tensor bytes
  + metadata for cross-machine forwarding
- InterDaemonEvent::MemoryPoolWrite: daemon↔daemon via Zenoh
- PROXY_POOL_DATA static: caches remote tensor data for local reads
- Handle incoming MemoryPoolWrite by storing in proxy pool

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Complete data path for cross-machine tensor transfer:
- DaemonRequest::WritePinnedMemory: node→daemon, carries tensor bytes
- DaemonNodeEvent::WriteMemoryPool: daemon handler, stores in PROXY_POOL_DATA
- InterDaemonEvent::MemoryPoolWrite: daemon↔daemon Zenoh forwarding
- DaemonReply::PinnedMemoryData: daemon→node, returns proxy pool data
- Control channel: hex-encode proxy data as Metadata with proxy_data key
- read_memory_pool: detect proxy_data, decode hex bytes, return as tensor
- write_memory_pool: serialize tensor after local write for cross-machine

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…forwarding

Remove the complex Zenoh publisher management from the WriteMemoryPool
handler.  The PROXY_POOL_DATA storage and read-path fallback are
complete and functional for same-machine cross-daemon testing.
Zenoh cross-daemon forwarding of InterDaemonEvent::MemoryPoolWrite
will be added in a follow-up PR.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
WriteMemoryPool handler now publishes InterDaemonEvent::MemoryPoolWrite
via a dataflow-global Zenoh topic (dora/{network}/{dataflow_id}/memory-pool).
All daemons subscribe to this topic at dataflow startup — incoming
events are deserialized and dispatched through the existing inter-daemon
event handler, which stores into PROXY_POOL_DATA for local reads.

- dataflow_memory_pool_topic(): new topic helper in dora-core
- spawn_dataflow(): subscribe to memory pool topic, spawn listener task
- WriteMemoryPool handler: publish via Zenoh for remote daemons

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… proxy path

Remote receivers rebuilt the proxied tensor as a raw uint8 view because
the WritePinnedMemory/MemoryPoolWrite chain only carried (bytes, size,
device). Add dtype/shape to every hop so the receiver reconstructs the
original tensor semantics:

- InterDaemonEvent::MemoryPoolWrite / DaemonReply::WritePinnedMemory:
  add dtype + shape fields
- WriteMemoryPool handler: store (bytes, size, device, dtype, shape) in
  PROXY_POOL_DATA via ProxyPoolEntry alias (fixes clippy type_complexity)
- Rust node API: write_pinned_memory() takes dtype/shape, exposed as
  Parameter::String/ListInt when reading proxy_data

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… on degraded links

- MemoryPoolWrite subscription listener moved out of the spawn handler:
  on a degraded inter-daemon link declare_subscriber() itself can block,
  wedging the daemon event loop (heartbeats + node replies included) —
  observed as the sender hanging on WritePinnedMemory forever
- publish offloaded to a tokio::spawn with CongestionControl::Block and
  explicit error logs: a dropped publish silently strands remote readers
  with a never-ready proxy pool (observed on WAN link hiccup mid-transfer)
- tcp listener: log frame size + first bytes when deserializing a
  DaemonRequest fails, so protocol drift is diagnosable

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ches

The new MemoryPoolWrite variant left replay-node and the record/echo/
hz/info commands with non-exhaustive matches (E0004). Add explicit arms:
replay-node and topic tools ignore the event (no-op/continue), matching
their handling of OutputClosed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…cross-machine receivers

register_memory_pool() now writes the registration tensor through
WritePinnedMemory so remote daemons receive it via Zenoh (CPU receivers
only; GPU pools travel via IPC handles, which the proxy path cannot
carry). Pulls dtype/shape from tensor info and logs push failures
loudly — a silent drop strands remote readers with a never-ready pool.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e-read

- sender: re-push registration data every 500ms until consumed; pace
  writes ~20s to outlast receiver read latency under host contention
- receiver: re-read (not zero-copy) each iteration — cross-machine
  proxy pools deliver fresh bytes per write

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e object header

Cross-machine receiver previews showed the PyBytesObject header
(refcount/type/len) instead of the tensor data: the dict's "ptr" used
PyBytes::as_ptr(), which yields the object start. Switch to
as_bytes().as_ptr() — the payload slice. Verified cross-machine
(5090↔A100): 61.44MB transfers now reconstruct byte-identical tensors
(sender preview == receiver preview).

Also clamp the peer-claimed size to the actual payload length: the CPU
tensor path builds (ctypes.c_byte * size).from_address(ptr), so an
inflated claim reads past the heap block. The local DORADMA and GPU
paths validate; the proxy path was the sole unguarded one.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two independent stalls kept the cross-machine example from completing
more than the first frame (verified on 5090↔A100 over a WAN):

1. daemon: bincode::serialize of the 61.44MB MemoryPoolWrite payload ran
   inline in the daemon event loop — 3.2s per frame in debug builds
   (hundreds of ms in release) — blocking output delivery (next_require)
   and subsequent node requests until the event channels backed up and
   the sender's WritePinnedMemory hung forever.  Move serialize +
   declare + put all into the spawned publish task.

2. sender.py: the trailing node.next() at the end of each iteration
   waited for the *next* iteration's next_require, which the receiver
   only sends after the *next* latency output — which this loop hasn't
   produced yet.  Classic self-deadlock: sender stuck at the second
   next() while the receiver waits for the next latency.  Drop it
   (keep the 20s pacing).

3. receiver.py: the memory-pool event trails the latency output on a WAN
   (separate topics, no ordering guarantee) and the registration re-push
   keeps old frames in the proxy pool — a read can return the previous
   frame (assert: expected 1, got 0).  Retry the read until the expected
   frame arrives (each read consumes one proxy entry).

Verified end-to-end: sender preview == receiver preview on all frames,
3-frame run completes with no errors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… perf_counter

time.perf_counter_ns() is CLOCK_MONOTONIC — its epoch is each machine's
boot time, so t_received - t_send across machines is dominated by the
boot-time difference (A100 up 34 days, 5090 up 5 hours → measured
0.00002 MB/s).  Both hosts are NTP-synced (same timezone, identical
wall-clock seconds), so time.time_ns() deltas are the true transfer
time: 12.94 MB/s measured over the WAN.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dual-end real DORADMA pools replace the proxy-pool + hex roundtrip
(11x gap: 12.94 vs 148 MB/s). register gains a `machine` param resolved
via the coordinator (warn-and-skip if unresolvable); write forwards the
full frame for the remote daemon to memcpy straight into the pre-registered
pool under the seqlock protocol; read stays the unchanged zero-copy fast
path; free tracks both ends. v1 scope: cpu2cpu_cross only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Strict reading of the requirement: when machine is specified but the
coordinator cannot resolve it (or there is no coordinator), the whole
register does nothing — no pool is created even locally — returns None
for the caller to check, and never crashes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolution failure and remote-creation failure now behave identically
(warn, no pool created, register returns None, no crash) — only the
warning text differs so the two failure classes are diagnosable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ees it

The synchronous register already guarantees the remote pool exists before
any write. Lazy creation on write is a redundant side path and a leak
source: a write-created pool is outside the free tracking (free events
only reference registered pools), so it would never be released. Missing
pool at write time is now a warn-and-drop-frame defensive case.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
8 tasks: message types, coordinator ResolveMachine (store API already
exists), daemon-A sync register with spawned ack wait (deadlock-free),
daemon-B pool mirror + direct seqlock writes + dual-end free, python
machine param, examples + local dual-daemon E2E + negatives, perf check.
Includes the daemon->coordinator runtime request-reply mechanism (new
pending-reply map + WS dispatch) needed by resolve_machine.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…achine

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e green)

The message-layer additions broke exhaustive matches in the coordinator,
node API, daemon and CLI/replay tools. Add stub arms (warn / not-yet-
implemented replies / no-op) so every commit compiles; T2-T4 fill in the
real implementations.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the T1 stub with the real store lookup
(get_daemon_by_machine); unknown machines resolve to found: false.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Distinguish a store failure from an unknown machine in the logs —
matches the codebase convention of warning on persistence errors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RegisterCrossMachinePool: resolve via coordinator, publish RegisterPool
over the memory-pool topic, await the remote RegisterPoolAck with a 5s
timeout in a spawned task (the ack arrives through the event loop, so
awaiting on the loop would deadlock). Warn texts differ for resolution
failure vs remote creation failure. Adds the daemon->coordinator
runtime request-reply mechanism (COORDINATOR_PENDING + WS dispatch).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The reply JSON nests found under the externally-tagged enum variant
("ResolveMachineResult"), so the previous extraction always returned
false and the successful register path was unreachable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CoordinatorSender::send_event wraps its own envelope with a fresh id,
so resolve_machine's pre-built envelope was double-wrapped and dropped
by the coordinator parse. Add send_event_with_id (single envelope with
the caller's request id) and send bare Timestamped bytes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-end free

RegisterPool creates a DORADMA pool mirror (same layout as the node
API) and acks; MemoryPoolWrite writes straight into the mirrored data
region under the seqlock protocol when the pool is cross-machine
(legacy proxy path unchanged otherwise); FreePool removes the mirror.
Extracts publish_memory_pool_event for the ack/free publishing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The publisher's own subscriber received its RegisterPool echo, failed to
mirror (EEXIST — the local node already created the pool) and published a
false ok=false ack that deterministically beat the remote's ack, failing
every sync register. Publish with Locality::Remote; gate RegisterPool on
the machine_id match; guard the direct write against corrupt headers;
move the 61.44MB memcpy off the event loop; init the mirror with an odd
generation so readers wait for the first write.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…machine mirror

machine=None (default) keeps the local path; machine="B" registers the
pool cross-machine through the daemon (coordinator resolve + sync ack).
Failure rolls back the local pool and returns None — never crashes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
register_cross_machine_pool transport errors went through ? — no local
rollback (leaking the shmem + host pin) and a Python exception, breaking
the warn-and-no-op contract. Merge both failure channels into the shared
rollback helper and return None.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When a cross-machine pool is freed, remove it from CROSS_POOLS and
publish FreePool so the peer releases the mirrored shmem (T4 handler).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
github-actions Bot and others added 25 commits August 12, 2026 22:24
Update JSON schema (9c29568)

Co-authored-by: Dora Bot <dora-bot@phil-opp.com>
…aflow-scoped cross state, subscriber lifecycle, non-Linux clippy)

- write_memory_pool now withholds its reply until the mirror daemon
  confirms the segment write (MemoryPoolWriteAck, seq-matched), so the
  send_output notification that follows the write can never overtake the
  tensor data and the receiver cannot return a stale frame; the example's
  300s polling workaround becomes unnecessary. Publish failures and a
  120s safety timeout fail the write loudly instead of hanging it.
- Cross-machine registration now rejects pools larger than
  MAX_MESSAGE_BYTES (64 MiB, 1 KiB margin for framing) with a clear
  error in both the daemon and the python extension — previously such
  pools registered fine but every per-frame push silently failed,
  leaving the receiver waiting forever.
- Cross-pool state (cross_pools, CROSS_REGISTER_PENDING) is keyed by
  (dataflow id, pool id) instead of pool id alone: every node process
  restarts its pool counter from zero, so a bare pool id repeats across
  concurrently running dataflows and could alias another flow's
  registration, ack routing, or free.
- The per-dataflow zenoh subscriber task handle is retained and aborted
  on finish_dataflow AND on the failed-spawn path (it is spawned before
  the node build, so a failed spawn never reached finish_dataflow),
  stopping the task/session/event-sender leak and duplicate consumers.
- cleanup_all keeps machine_id used on non-Linux (clippy -D warnings).

Tests: dora-memory-pool 14/14 (incl. new cross_pool_state_is_dataflow_scoped),
dora-daemon --lib 209/209; clippy -D warnings clean on daemon/memory-pool/
message/cli; fmt clean. The python-extension clippy lint errors are
pre-existing (pyo3 deprecations/unsafe blocks, excluded from CI clippy).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The new variant was missing from replay-node's exhaustive match (CI
Check + Clippy both failed on the same E0004).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ternative)

The node's cross-machine write request now carries only (id, size)
metadata; the daemon reads the tensor from the sender's segment (name
resolved deterministically first — the register-time initial push
arrives before the python's local registration lands — then the daemon
table for explicit name= pools) and forwards it through the existing
zenoh + commit-ack path. The node→daemon request is KB-scale, so the
MAX_MESSAGE_BYTES (64 MiB) transport cap no longer applies: pools up to
the 1 GiB registration cap transfer correctly, and the registration-time
rejection added earlier is removed.

Errors reply to the node instead of propagating: a handler error tore
down the node connection and cascaded into a daemon disconnect
(observed: 'pool X has no local segment to read the write from' killed
the WS connection, and the reconnect's startup sweep then removed the
just-created segment).

Verified: same-host cross-daemon smoke (torch-gated) passes end-to-end
with the new path; dora-memory-pool 14/14, dora-daemon --lib 209/209,
clippy/fmt clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- The same-host cross-daemon smoke reads via the direct==true path and
  bypasses the MemoryPoolWrite/MemoryPoolWriteAck machinery entirely
  (only manual two-host runs exercised it), so the ack resolution is
  extracted into resolve_cross_write_ack() and pinned by a unit test:
  a stale seq resolves nothing, the seq-matched ack resolves exactly
  its own pending reply, and a failed mirror write surfaces as an error
  reply.
- The read fast-path bail message now reports the actual wait window
  (0.5s for local pools) instead of the hardcoded 3600s.

dora-daemon --lib 210/210, dora-memory-pool 14/14, clippy/fmt clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Covers the multi-daemon bring-up (coordinator + --machine-id daemons,
--local-listen-port on one host, zenoh rendezvous), the YAML essentials
(cross_machine env, _unstable_deploy machine/working_dir), the true-WAN
ZENOH_CONFIG three points, the commit-ack and shmem-reference write
semantics, measured numbers (LAN ~40, WAN ~4 MB/s; native ≤1 MiB on WAN,
ROS 2 RTT-paced), and a cross-machine debugging checklist.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- The shared-memory-reference read (full-size allocation + copy) now
  runs inside the spawned publish task instead of synchronously on the
  daemon event loop — a 61.44MB frame previously blocked heartbeats,
  node replies, and output delivery for the duration of the read.
  Errors resolve the pending write reply (seq-matched) as before.
- The mirror-creating daemon now enforces the same 1 GiB cap as the
  local side: create_cross_pool_shmem allocates size + data_offset in
  /dev/shm straight from the remote RegisterPool event, so a buggy or
  corrupted peer could previously drive an unbounded allocation
  (memory-exhaustion DoS). The error flows back through RegisterPoolAck.
- The mirror header JSON is built with serde_json instead of format!
  interpolation: dtype/device arrive from the remote event (untrusted
  strings) and quotes/backslashes could corrupt or inject into the
  parsed structure.

dora-daemon --lib 210/210, dora-memory-pool 14/14, clippy/fmt clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The README describes usage and behavior; measured throughput lives in
design.md §5 (single source of truth, updated with the 2026-08-11
LAN/WAN runs).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cross-machine writes now bypass the zenoh relay when the mirror daemon
advertises a data listener:

- The mirror daemon runs a direct-TCP listener (port 7410, overridable
  via DORA_MEMORY_POOL_DATA_PORT) and reports it in RegisterPoolAck.
- The origin learns the target daemon's address from the coordinator
  (ResolveMachine now returns the target's WS peer address, tracked at
  registration) and opens a persistent connection per endpoint.
- Frames carry [magic][dataflow][pool][seq][size][data]; the mirror
  reads the payload straight into the mirror segment's data region under
  the per-pool async lock + seqlock (zero user-space copies on the
  receive side); the origin pays a single user-space copy (segment →
  send buffer). The commit ack still arrives via zenoh, so the pending
  machinery is unchanged.
- Falls back to the zenoh relay when no endpoint is known or the direct
  send fails (dead connection dropped and lazily re-established).

dora-daemon 210/210, dora-coordinator 122/122, dora-memory-pool 14/14,
clippy -D warnings and fmt clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…P codec

- The direct-TCP data listener's accept loop now sleeps 50ms on accept
  errors — a persistent error (EMFILE/ENFILE fd exhaustion) previously
  spun at 100% CPU.
- serve_cross_data_frame is split into handle_cross_data_frame (frame
  parse + mirror write, no zenoh) and the zenoh ack publish, and the
  codec is pinned by a loopback round-trip test: send_cross_data_frame
  → handle_cross_data_frame over a TcpListener, asserting the payload
  lands in the mirror's data region under an even seqlock generation and
  the returned ack info matches (dataflow, pool, seq). The same-host
  smoke (direct == true) bypasses this data plane, so this is the new
  steady-state write path's first automated coverage.

dora-daemon 211/211, clippy/fmt clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The data listener previously bound at daemon startup for every daemon
with a machine id, even ones that never mirror a cross-machine pool.
It now opens only when the first RegisterPool asks this daemon to
mirror something (in the RegisterPool handler, after the machine gate),
so non-participating daemons never open the port. The bound port is
still advertised in RegisterPoolAck.data_port.

dora-daemon 211/211, clippy/fmt clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The origin dials the mirror daemon's coordinator-visible WS source
address, which is the wrong dial target under NAT, multi-homed, or
same-host coordinator deployment (e.g. 127.0.0.1 when the daemon
connects to a co-located coordinator) — the direct fast path would
silently never engage there. The mirror daemon now advertises an
explicit dialable address via DORA_MEMORY_POOL_DATA_ADDR (full
ip:port, parsed with a warn on garbage), carried in
RegisterPoolAck.data_addr; the origin prefers it over the derived
address, falling back to the derived one otherwise.

dora-daemon 211/211, clippy/fmt clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The write-commit protocol's last un-covered link was the ack publish
itself: the resolver was pinned by a unit test, but the mirror-side
publish -> zenoh transport -> origin-side subscribe -> deserialize
chain was only exercised by manual two-host runs. The new test spins
up two hermetic zenoh sessions (loopback TCP, no scouting) and drives
the production publish helper (Locality::Remote, so a same-session
subscriber would never see the put), receives the ack on the origin
subscriber, deserializes with the production method, and asserts the
seq-matched pending reply is resolved.

Also removes a duplicated doc comment on serve_cross_data_frame.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The direct-TCP → zenoh fallback is the steady state on a broken link, so
the per-frame warn flooded the daemon log for the whole outage. The
fallback is now tracked per pool (CROSS_DIRECT_DEGRADED, keyed like
CROSS_WRITE_PENDING): the first failed write warns with the error,
subsequent failures stay silent, and the first successful write after a
fallback logs recovery once. Pinned by a unit test on the note_*
state machine.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… write errors

Two direct-TCP data-plane issues from review:

1. `data_offset + size` was an unchecked add on the wire-controlled
   `size` (u64 read straight off the socket): a size near u64::MAX
   wraps the sum past the bounds check, and a size-byte slice would be
   constructed past the mapping (UB; a remote-triggerable abort in debug
   builds). All three `data_offset + len` guards now use checked_add
   (handle_cross_data_frame, write_cross_pool_data,
   read_pool_segment_data), with the frame path rejecting with the ack
   info.

2. A mid-frame payload read failure left the mirror seqlock odd and
   published no ack, stranding the origin until the 120s commit-ack
   timeout. handle_cross_data_frame now returns CrossFrameError
   carrying the frame's (dataflow, pool, seq) once the header is
   parsed, and serve_cross_data_frame publishes MemoryPoolWriteAck
   { ok: false } on that path so the origin fails fast. The odd
   generation is kept deliberately (fail-safe: readers reject the torn
   frame; the next full write self-heals) — rolling it back would mark
   half-written bytes as a complete frame.

Tests: wire_size_overflow_is_rejected_not_aborted (u64::MAX size must
be rejected with ack info, generation untouched),
payload_read_failure_stays_odd_and_carries_ack (mid-frame drop leaves
the odd generation, carries ack info, next write self-heals).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The inline comment described a Drop-rollback of the seqlock generation
that no longer exists: the generation deliberately stays odd (fail-safe,
readers reject the torn frame, next write self-heals) — matching the
NOTE on DirectMirrorWriter and the payload_read_failure_stays_odd test.
The old wording would have misled a future maintainer into assuming
rollback safety that is not there.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ross-write state

The per-pool async write lock was keyed by the bare pool id. Pool ids
repeat across dataflows (each node process restarts its counter), so two
concurrent dataflows writing their own 'pool_sender_node_1' segments
shared one lock and were needlessly serialized — the same key-scoping
issue the human review flagged for cross_pools. The lock is now keyed by
(dataflow, pool): writes to one segment stay serialized, writes to
different segments of concurrent dataflows run in parallel. Regression
test write_lock_is_dataflow_scoped pins the aliasing case.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-keying the direct-write lock map by (dataflow, pool) fixed the
aliasing but changed the growth bound: the old bare pool-id key was
bounded (pool ids repeat across dataflows), the new per-dataflow-UUID
key accumulates one entry per (dataflow, pool) for the daemon's
lifetime — a slow unbounded leak on a long-lived daemon cycling many
cross-machine dataflows. finish_dataflow now drains the finished
dataflow's entries from the async write-lock map, the write-seq
counters (same never-drained shape, predates this PR), and the
degradation set.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The direct-TCP data plane's error path (serve publishes
MemoryPoolWriteAck { ok: false } when a frame's identity is parsed but
the write cannot proceed) had no automated coverage — the codec
round-trip covered the happy path, and the error path's zenoh ack
publish was only exercised by manual runs. The new test drives
serve_cross_data_frame with a size-overflow frame over a real zenoh
session pair (hermetic loopback, no scouting) and asserts the origin
subscriber receives ok=false with the matching (dataflow, pool, seq)
and the overflow error. This closes the last testable half of the
standing two-host coverage non-blocker; only the true two-host
transfer remains manual.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cargo-audit fails CI on the new advisory (2026-07-29): Unix BROWSER
handling in webbrowser 1.2.1 allows browser argument injection. Bump to
1.2.4, which pulls in the fixed release plus its updated transitive
dependencies (objc2-app-kit etc.).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The unpinned install-action 'latest' raced between runner caches: one of
the two parallel CI runs per push picked up an older cargo-audit that
crashes parsing the Cargo.lock's multi-target windows-sys (webbrowser's
windows-only 0.48.0 target dependency — 'invalid Cargo.lock dependency
tree: failed to find dependency: windows-sys 0.48.0'), while the other
run's newer cargo-audit passed. Verified locally: cargo-audit 0.22.2
parses the lock cleanly and reports the webbrowser advisory resolved.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
install-action's tool@version form did not pin the runner's cached
cargo-audit, so the parallel CI runs raced: the runner with a stale
cached release crashed parsing the Cargo.lock's multi-target
windows-sys (webbrowser's windows-only 0.48.0 dependency), the runner
without it passed. cargo install --version 0.22.2 removes the cache
from the equation; 0.22.2 parses the lock cleanly (verified locally
against the exact HEAD lockfile).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tang-canran and others added 2 commits August 12, 2026 22:32
- daemon/src/coordinator.rs: keep our ResolveMachine reply routing
  (ReplyRouteRaw + resolve_machine) — upstream does not have the
  cross-machine ResolveMachine command; their side of the conflict was
  comment wording only
- apis/rust/node/src/node/mod.rs, runtime-api: take upstream's newer
  versions (our branch only carried synced upstream commits)
- runtime/src/operator/mod.rs: take upstream's deletion (runtime
  refactor; memory-pool does not touch it)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Merging upstream/main overwrote apis/rust/node/src/node/mod.rs with the
upstream version, dropping the write_pinned_memory and
register_cross_machine_pool wrappers our python extension calls (the
control_channel protocol methods survived the merge). Restored both
wrappers after free_pinned_memory. Lower unwrap budget 162 -> 160
(upstream main removed unwraps).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copy link
Copy Markdown
Collaborator

🤖 Automated review by Claude — this review is fully automated (no human wrote it) and may contain mistakes; please verify before acting.

Follow-up on the two commits that landed since the last automated pass: 6549d011 (merge of upstream/main) and b12cee9b (restore lost node-API methods).

  • The restore in apis/rust/node/src/node/mod.rs re-adds write_pinned_memory and register_cross_machine_pool as thin delegations to the matching control_channel methods. Both signatures match the control-channel implementations, so the restore is correct and the Python extension's call sites are satisfied again.
  • No new correctness issues in this delta. One process note, not a blocker: the merge commit is large (~6k lines of upstream changes) and, by its own message, resolved conflicts in coordinator.rs, node/mod.rs, runtime-api, and runtime/src/operator/mod.rs by taking one side or the other. The fact that it silently dropped the two wrappers (caught and fixed in the follow-up) shows the merge needs CI to confirm no other memory-pool surface was lost — worth a green full build/test before merge rather than trusting the conflict resolution alone.

Previously-reported items all appear addressed in earlier rounds; the only standing non-blocker remains the true two-host cross-machine path, which can't be exercised in single-host CI by physical constraint.


Generated by Claude Code

@tang-canran

Copy link
Copy Markdown
Contributor Author

Both points verified — the restore is confirmed, and the process note is addressed with a mechanical audit rather than trusting the conflict resolution:

1. Pub-symbol audit across the merge (491dffcfb12cee9b). For every memory-pool-touching file I compared the exported pub fn/struct/enum/trait sets on both sides of the merge (comm -23 of the sorted symbol lists):

  • apis/rust/node/src/node/mod.rs — the only lost symbols are the two wrappers (write_pinned_memory, register_cross_machine_pool) already restored in b12cee9b
  • apis/python/node/src/lib.rs, libraries/message/src/memory_pool.rs, libraries/message/src/daemon_to_node.rs, libraries/core/src/topics.rs, binaries/daemon/src/lib.rs, binaries/coordinator/src/lib.rs, binaries/coordinator/src/state.rs, binaries/daemon/src/spawn/spawner.rs, binaries/replay-node/src/main.rs, runtime/src/operator/mod.rs, runtime-api — zero lost symbols
  • libraries/extensions/memory-pool was not touched by the merge at all
  • The only pool-related "removal" in the merge delta (_register_host call in apis/python/node/src/lib.rs) is a let-chain refactor, not a loss

2. Green full build/test at head. CI run on b12cee9b (31607758462) is fully green: Check (full build + unit tests), Clippy, Format, Audit (cargo-audit + cargo-deny), Typos, Unwrap budget, License, GitGuardian, find_package smoke all pass.

The standing non-blocker (true two-host cross-machine path) remains as-is — verified on the real WAN cluster (direct-TCP 51.15 MB/s LAN / 9.99 MB/s WAN, full-byte parity 30/30), but stays outside single-host CI by physical constraint.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants